[Feature] Sequence sample unit with exact boundary policies - #4050

…hor priority semantics

Pieces 3 and 4 (first half) of the pytorch#4039 split, on top of the Sequence
unit from pytorch#4050. The window around each anchor becomes burn_in records
before the anchor, the learning region of length records starting at
it, and bootstrap records after it, with stride spacing the whole
window uniformly. A per-record learning_mask info entry is True
exactly on the learning region. Burn-in never shifts the anchor:
entries before the episode start are invalid and clamp to it;
bootstrap entries obey the episode_boundary policy at episode ends;
defaults reproduce the previous behavior exactly.

Priorities live per anchor: a per-record anchor_index info entry
reports the storage index of each record's sampled anchor (the
original anchor, not the stop-shifted one, since that is what the
sampler's distribution selected), so priorities of sampled sequences
update through the ordinary update_priority path, and per-anchor
sampler entries such as importance weights expand block-constant
across the window. Seeded distribution tests pin that range expansion
does not bias anchor selection for uniform or prioritized sampling.

Part of pytorch#4039.

…hor priority semantics

Pieces 3 and 4 (first half) of the pytorch#4039 split, on top of the Sequence
unit from pytorch#4050. The window around each anchor becomes burn_in records
before the anchor, the learning region of length records starting at
it, and bootstrap records after it, with stride spacing the whole
window uniformly. A per-record learning_mask info entry is True
exactly on the learning region. Burn-in never shifts the anchor:
entries before the episode start are invalid and clamp to it;
bootstrap entries obey the episode_boundary policy at episode ends;
defaults reproduce the previous behavior exactly.

Priorities live per anchor: a per-record anchor_index info entry
reports the storage index of each record's sampled anchor (the
original anchor, not the stop-shifted one, since that is what the
sampler's distribution selected), so priorities of sampled sequences
update through the ordinary update_priority path, and per-anchor
sampler entries such as importance weights expand block-constant
across the window. Seeded distribution tests pin that range expansion
does not bias anchor selection for uniform or prioritized sampling.

Part of pytorch#4039.

…hor priority semantics

Pieces 3 and 4 (first half) of the pytorch#4039 split, on top of the Sequence
unit from pytorch#4050. The window around each anchor becomes burn_in records
before the anchor, the learning region of length records starting at
it, and bootstrap records after it, with stride spacing the whole
window uniformly. A per-record learning_mask info entry is True
exactly on the learning region. Burn-in never shifts the anchor:
entries before the episode start are invalid and clamp to it;
bootstrap entries obey the episode_boundary policy at episode ends;
defaults reproduce the previous behavior exactly.

Priorities live per anchor: a per-record anchor_index info entry
reports the storage index of each record's sampled anchor (the
original anchor, not the stop-shifted one, since that is what the
sampler's distribution selected), so priorities of sampled sequences
update through the ordinary update_priority path, and per-anchor
sampler entries such as importance weights expand block-constant
across the window. Seeded distribution tests pin that range expansion
does not bias anchor selection for uniform or prioritized sampling.

Part of pytorch#4039.

…hor priority semantics

Pieces 3 and 4 (first half) of the pytorch#4039 split, on top of the Sequence
unit from pytorch#4050. The window around each anchor becomes burn_in records
before the anchor, the learning region of length records starting at
it, and bootstrap records after it, with stride spacing the whole
window uniformly. A per-record learning_mask info entry is True
exactly on the learning region. Burn-in never shifts the anchor:
entries before the episode start are invalid and clamp to it;
bootstrap entries obey the episode_boundary policy at episode ends;
defaults reproduce the previous behavior exactly.

Priorities live per anchor: a per-record anchor_index info entry
reports the storage index of each record's sampled anchor (the
original anchor, not the stop-shifted one, since that is what the
sampler's distribution selected), so priorities of sampled sequences
update through the ordinary update_priority path, and per-anchor
sampler entries such as importance weights expand block-constant
across the window. Seeded distribution tests pin that range expansion
does not bias anchor selection for uniform or prioritized sampling.

Part of pytorch#4039.